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Abstract 

The  fail-stop  failure  model  appears  frequently  in  the  distributed  systems  literature.  How¬ 
ever,  in  an  asyndironous  distributed  system,  the  fail-stop  model  cannot  be  implemented.  In 
particular,  it  is  impossible  to  reliably  detect  crash  failures  in  an  asynchronous  system. 

In  this  paper,  we  show  that  it  is  possible  to  specify  and  implement  a  failure  model  that  is 
indistinguishable  from  the  fail-stop  model  bom  the  point  of  view  of  any  process  within  an 
asynchronous  system.  We  give  necessary  conditions  for  a  failure  model  to  be  indistinguishable 
from  the  fail-stop  model,  and  derive  lower  bounds  on  the  amount  of  process  replication  needed 
to  implement  such  a  failure  model.  We  present  a  simple  one-round  protocol  for  implementing 
one  sudi  failure  model,  which  we  call  simulated  fail-stop. 


1  Introduction 

Uie  fail-stop  failure  model  appears  frequently  in  the  distributed  systems  literature.  The  fail-stop 
model  makes  two  assumptions  about  the  failure  behavior  of  processes:  processes  fail  only  by 
permanently  crashing  and  when  a  process  crashes,  surviving  processes  will  eventually  detect 
that  failure.  The  fail-stop  model  is  appealing  because  it  makes  distributed  algorithms  easier  to 
formulate:  fail-stop  failures  are  easy  to  tolerate. 

For  example,  suppose  that  a  set  of  processes  {1, 2, n}  wish  to  solve  iheelection  problem:  at  any 
p(^t,  no  more  than  one  process  of  the  set  can  be  the  leader,  and  as  long  as  all  processes  do  not  foil, 
it  is  always  the  case  that  there  will  eventually  be  a  leader.  Assuming  a  foil-stop  foilure  model  leads 
to  a  very  simple  solution.  Each  process  maintains  a  local  copy  of  the  list  (1, 2, ...,  n),  and  the  first 
element  of  this  list  denotes  the  leader.  When  process  t  detects  the  foilure  of  process  j,  i  removes  j 
boms  its  local  copy  of  the  list.  When  i  finds  itself  the  first  element  of  its  list,  t  knows  that  it  is  the 
leader.  Since  a  process  becomes  the  head  of  the  list  only  when  all  lower-numbered  processes  have 
failed,  there  is  no  more  than  one  leader  at  any  time;  and,  as  long  as  a  process  eventually  detects 
the  foilxue  of  the  lower-numbered  processes,  it  will  eventually  become  the  leader. 

*This  work  was  supported  by  the  Defense  Advanced  Research  Projects  Agency  (DoD)  under  NASA  Ames  grant 
number  NAG  2-593,and  by  grants  bom  IBM  and  Siemens.  The  views,  opinions,  and  findings  contained  in  diis  report 
are  diose  of  the  authors  and  should  not  be  construed  as  an  official  Department  of  Defense  position,  p(4icy,  or  decision. 

*This  author  is  also  suppmted  by  an  AT&T  PhD  Schcdar^p. 
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A  serious  limitation  of  assuming  a  feiil-stop  hulure  model  is  that  it  is  often  an  unrealistic 
assumption.  In  particular,  in  an  asynchronous  distributed  system  (i.e.,  a  system  with  no  shared 
memory,  arbitrary  message  delivery  times,  no  global  dock,  and  arbitrary  process  speeds),  the 
fail-stop  model  cannot  be  implemented.  This  is  because  it  is  impossible  to  reliably  detect  crash 
failures  in  an  asynchronous  sy^em  (see  Theorem  1). 

On  the  other  hand,  there  are  systems  (e.g.,  ISIS  [BJ87])  that  provide  crash-failure  detection  with¬ 
out  making  synchrony  assumptions.  They  do  this  by  allowing  failures  to  be  detected  erroneously, 
e.g.,  by  using  timeouts  and  gossip  messages  ([RB91])  to  attain  agreement  among  a  set  of  processes 
that  a  process  p  has  holed  even  though  that  process  p  may  not  have  crashed.  Hence,  they  provide 
a  ^ure  model  that  resembles  fail-stop  in  some  ways  but  is  not  strictly  fail-stop. 

In  this  paper,  we  present  a  hiilure  model,  called  simulated  fail-stop,  that  is  internally  indistin¬ 
guishable  from  ^-stop,  meaning  that  under  this  model,  no  process  in  the  system  can  determine 
that  it  is  not  running  in  a  system  in  which  the  fail-stop  assumption  holds.  We  give  a  set  of  con¬ 
ditions  that  are  necessary  in  order  for  any  model  to  be  indistinguishable  hnom  hul-stop,  and  we 
prove  that  simulated  fail-stop  is  indistinguishable  from  fail-stop.  We  give  lower  bounds  on  the 
number  of  processes  needed  for  a  one-round  implementation  of  the  simulated  fcul-stop  model  to 
tolerate  i  failures,  and  show  that  these  bounds  hold  for  any  model  that  is  indistinguishable  from 
foil-stop.  Finally,  we  show  that  the  bounds  are  ti^t  by  giving  a  protocol  that  attains  them. 

The  paper  is  organized  as  follows.  Section  2  describes  the  system  model  used  throughout 
the  paper,  including  notation,  definitions,  and  a  formal  logic  used  to  describe  system  properties. 
Section  3  specifies  the  foil-stop  and  simulated  fail-stop  models,  introduces  the  notion  of  indistin- 
guishability  of  foilure  models,  and  proves  that  certain  conditions  are  necessary  and/or  sufficient 
for  a  failure  model  to  be  indistinguishable  from  fail-stop.  Section  4  gives  lower  bounds  on  the 
number  of  processes  needed  to  tolerate  t  failures  for  one-round  failure  detection  protocols  imple¬ 
menting  the  simulated  fail-stop  model,  and  shows  that  these  bounds  hold  for  any  model  that  is 
indistinguishable  fix)m  foil-stop.  Section  5  shows  that  these  lower  bounds  are  tight  by  presenting 
a  protocol  that  meets  them.  Section  6  concludes  the  paper  and  discusses  the  work  that  remains  to 
be  done  on  this  topic. 


2  System  Model 

We  consider  a  distributed  system  consisting  of  a  set  of  n  processes  P  =  {1,2, A  process 
foils  by  simply  stopping  execution  (crashing),  and  a  failed  process  does  not  recover.  The  ^stem 
is  asynchronous,  meaning  that  the  rate  of  execution  of  any  process  with  respect  to  any  other  is 
unbounded  and  there  are  no  phyacal  clocks.  Between  any  two  processes  t  and  j  there  exist  two 
unidirectional  FIFO  channels:  fix>m  t  to  j  and  from  j  to  I.  Processes  communicate  only 

by  sending  and  receiving  messages  over  these  channels.  The  diannels  are  nonfoulty:  they  do 
not  lose,  generate,  or  garble  messages.  Message  delivery  time  is  unbounded.  We  assume  for 
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simplicity  that  channels  have  infinite  buffers  and  that  all  messages  m  are  unique  (they  can  easily 
be  made  so  by  including  in  m  its  source  and  a  sequence  number).  The  state  of  a  channel  is  the 
sequence  of  messages  that  have  been  sent  cdong  the  channel  but  not  received  along  the  channel. 

A  process  is  defined  by  a  set  of  states,  one  of  which  is  denoted  the  initial  state.  The  state  of 
a  process  i  consists  of  the  values  of  all  internal  variables  of  the  process,  plus  the  values  of  n  +  1 
additional  boolean  variables  that  are  defined  as  follows: 

•  crash{.  This  variable  is  initially  false  and  can  become  true  at  any  time.  Once  crashi  becomes 
true,  the  state  of  t  does  not  change  further.  (This  models  the  failure  of  i.) 

•  Vj  €  P:  failedi(j).  This  variable  is  initially  false  for  all  values  of  j,  and  becomes  true  when  i 
detects  the  crash  of  process  j.  Once  failed,-(j)  becomes  true,  it  remains  true  forever.  Exactly 
when  failedi(j)  becomes  true  with  respect  to  when  crashj  becomes  true  is  discussed  in  this 
paper. 

A  global  state  of  the  system  is  a  set  of  process  and  channel  states.  An  initial  global  state  is  the 
global  state  in  which  each  process  state  is  an  initial  state  and  each  channel  state  is  the  empty 
sequence. 

An  event  e  is  a  function  that  maps  global  states  to  global  states.  An  event  e  applied  to  a  global 
state  S  yields  a  new  global  state  S'  that  differs  fit>m  S  in  the  local  state  of  exactly  one  process  t 
and  the  state  of  at  most  one  channel  incident  on  i.  We  say  in  this  case  that  e  is  an  event  of  i,  and 
that  c  changes  the  state  of  i. 

If  an  event  e  of  process  t  changes  the  state  of  C,  j  for  some  j,  then  we  call  e  a  send  event  A  send 
event  changes  the  state  of  a  channel  by  appending  a  message  m  to  the  sequence  of  messages  on 
that  chaimel.  If  e  changes  the  state  of  for  some  j,  then  we  call  e  a  receive  event.  A  receive  event 
changes  the  state  of  a  channel  by  removing  a  message  hrom  the  head  of  the  sequence  of  messages 
on  that  chaimel. 

We  define  events,  runs,  and  predicates  formally  in  Appendix  A.l.  Informally,  send,  receive, 
crash,  and  failure  detection  events  are  defined  as  follows: 

•  sendi{j,  m)  denotes  the  event  whereby  process  i  sends  the  message  m  to  process  j. 

•  recviij,  m)  denotes  the  event  whereby  process  i  receives  the  message  m  from  process  j.  _ 

For 

•  crashi  denotes  the  event  whereby  crash,  becomes  true.  i 

•  ^iledj(j)denotestheeventwherebyfailedi(i)  becomes  true. 

Definition  1  Arvaiof  the  system  is  an  infinite  sequence  of  global  states  of  the  sif stem:  r  =  (So,  Si,  Sj, . . .), 
where  Sq  is  an  initial  global  stateand  there  exists  a  sequence  of  events  {cq,  ei,  ez, . . .)  such  that  for  all  i  >  0, 

s,+i  =  c.(s,).  i 


Definition  2  Given  any  run  r  =  (Eq,  Si,  S2,  • . .)/  history  ofr,  denoted  Hr,  is  the  sequence  of  events 
(eo,  ei,  ej, . . .)  such  that  for  all  i  >  0,  =  ej(  Ej). 

Note  that  for  any  run  r,  Tir  is  uniquely  determined.  Furthermore,  r  can  be  constructed  ftom  a 
history  Hr  and  the  initial  global  state  Eq. 

Throughout  this  paper,  we  use  the  notation  Hr  =  (•  •  •  Cj  •  •  •  e,  •  -  •  e*  •  •  •).  This  denotes  that  Tf, 
is  of  the  form  (i;  ea  y;  ej;  2;  e*;  w),  where  e„  Cj,  and  Ck  are  events,  i,  y,  and  z  are  finite  sequences  of 
events,  and  w  is  an  infinite  sequence  of  events. 

We  specify  properties  of  systems  using  predicate  logic  over  global  states  and  linear-time 
temporal  logic  over  (infinite)  suffixes  of  runs  [PneTT].  We  define  the  boolean  predicates  SEND^  ( j,  m ) 
and  RECV,(  j,  m)  as  foUows. 

•  Vi,  j,  m :  SEND,(i,  m)  and  RECVj(j,  m)  are  false  in  an  initial  global  state. 

•  sendiiJ,  "*)(S)  N  SENDj(j,  m).  That  is,  SENDi(i,  m)  becomes  true  when  sendi(j,  m)  has  oc¬ 
curred. 

•  recvi(j,  m)(E)  ^  RECV,(j,  m).  That  is,  RECV,(j,  m)  becomes  true  when  nrcn,(j,  m)  has  oc¬ 
curred. 

Furthermore,  both  SEND,(j,  m)  and  RECV<(j,  m)  are  stable  by  definition:  once  such  a  predicate 
becomes  true  in  a  run,  it  remains  true  for  the  remainder  of  the  run.  (ICL85]) 

We  define  the  boolean  predicates  CRASHj  and  failed, (j)  as  follows.  Let  E  be  a  global  state. 

•  E  ^  CRASH,  if  and  only  if  crash,  is  true  in  E. 

•  Vj :  E  1=  FAILED<(  j)  if  and  only  if  failed{(i)  is  true  in  E. 

Note  that  both  CRASHi  and  failed,  (  j)  are  stable  by  assumption:  once  these  local  variables  become 
true  in  the  local  state  of :,  they  remain  true  thereafter. 

Let  s  =  (So,  Si,  S2, . . .)  be  a  suffix  of  a  run,  let  v?  be  a  predicate,  and  let  7^  be  a  temporal  logic 
finmula. 

•  {s,k)  t=  <,pi£f  Sjfc  \=  (f. 

.  (s,A:)h0^iff3i>fc:  {s,j)\=V 

•  {s,k)^OViH'ij>k:  (s,j)|=7> 

Furthermore,  we  abbreviate  (r,  0)  ^  T’  as  r  |=  P. 

We  define  the  failed-before  relation  as  follows: 

Definition  3  //  r  [=  OFAILEDj(i)  in  some  run  r,  we  say  that  i  failed  before  j  in  r. 
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Note  that  it  is  possible  that  both  crash,  and  crash^  hold  in  some  global  state  yet  neither  i  foiled 
before  j  nor  j  foiled  before  i. 

We  use  a  version  of  the  happens  before  relation  of  (Lam78].  Given  two  events  e\  and  e2,  define 
ei  —  €2  (read  "ei  happens  before  ej")  in  some  history  'Hr  if  one  of  the  three  following  conditions 
holds: 

1.  e\  and  €2  are  of  the  same  process,  and  either  ei  =  62  or  ei  precedes  €2  in  Hr  ', 

2.  ei  =  sendi{j^  m)  for  some  value  of  i,j,  and  m,  and  €2  =  recvj{i,  m); 

3.  there  exists  an  event  e  such  that  e  and  e  —  ea- 

The  happei\s-before  relation  as  defined  here  is  the  same  a.~  that  given  in  [LamTSl,  except  that 
our  relation  is  reflexive.  This  is  for  notatioiuil  convenience.  Note  that  for  all  ei  ^  C2/  ci  — ^  e2 
implies  that  ei  precedes  €2  in  Hr-  The  converse  does  not  hold,  however. 

Let  r  be  a  run.  Let  r,  be  the  sequence  of  states  of  i  in  r,  with  repeated  states  removed  (i.e.,  so 
that  adjacent  states  are  distinct).  If  x  and  y  are  runs,  then  we  say  that  run  x  is  isomorphic  to  run  y 
with  respect  to  process  i,  denoted  x  y,  if  and  oidy  if  x,-  =  y,.  In  other  words,  x  y  if  and  only 
if  runs  x  and  y  are  indistinguishable  to  process  t.  Similarly,  for  Q  C  P  is  the  sequence  of  states 
of  processes  i  e  Q  in  r  with  repeated  states  removed,  and  x  y  if  and  only  if  xq  =  yg.  (See 
{GM86]  for  a  detailed  discussion  of  the  ramifications  of  indistinguishability  of  runs.) 

3  Specification  of  Failure  Models 

A  foilure  model  describes  the  manner  in  which  the  components  of  a  system  can  foil.  For  our 
purposes,  a  foilrire  model  constrains  how  crash  events  and  failed  events  can  occur  with  respect  to 
each  other.  We  give  these  constraints  as  a  set  of  properties  and  define  the  foilure  model  as  the  set 
of  runs  that  satisfy  these  properties. 

3.1  The  Fail-Stop  Failure  Model 

The  minimal  set  of  foil-stop  assumptions  fovmd  in  the  literature  is  that  in  any  infinite  run  of  the 
system,  a  process's  foilure  is  eventually  detected  by  all  processes  that  don't  crash,  and  that  there 
are  no  folse  detections  of  failure.  These  two  conditions  ^>ecify  the  foilure  model  defined  in  [Sch84]. 
Hence,  we  adopt  this  as  the  definition  of  the  fail-stop  foilure  model 
Formally,  the  two  foil-stop  conditions  are: 

FSl:  Vr,  i:  r  |=  □(CRASHi  =$►  Vj:  0(CRASHj  V  FAILED, (j))) 

FS2:Vr,i,j:  r  f=  □(FAILED, (i)=>-CRASH,) 

We  denote  with  FS  the  set  of  runs  satisfying  properties  FSl  and  FS2. 
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Theorem  1  In  an  asynchronous  system  in  which  crash  failures  are  possible,  properties  FSl  and  FS2  are 
impossible  to  implement. 

Proof  :  In  [CT91],  an  algorithm  is  given  for  solving  Consensus  with  a  Strong  Failure  Detector.  A 
Strong  Failure  Detector  is  shown  to  be  strictly  weaker  than  a  Perfect  Failure  Detector,  implying 
that  a  Perf^  Failure  Detector  can  also  be  used  to  solve  Consensus.  A  solution  to  Consensus 
contradicts  the  result  of  [FLP85];  therefore,  a  Perfect  Failure  Detector  cannot  be  constructed. 

A  Perfect  Failure  Detector  is  defined  in  [CT91]  as  a  failure  detector  satisfying  Strong  Com¬ 
pleteness  and  Strong  Accuracy.  These  two  properties  are  identical  to  FSl  and  FS2.  Therefore, 
implementing  FS  is  equivalent  to  implementing  a  Perfect  Failure  Detector,  and  is  therefore  im¬ 
possible.  □ 

3^  Indistinguishable  Failure  Models 

A  process  determines  which  event  to  execute  based  on  its  state  and  the  messages  that  it  has 
received.  A  run  r  is  isomorphic  to  a  run  r'  with  respect  to  a  process  i  if  i  executes  the  same  events 
in  both  r  and  r'.  We  know  that  the  two  runs  are  isomorphic  with  respect  tot  if  t  starts  in  the  same 
initial  state  in  both  runs,  receives  the  same  messages  in  the  same  order  in  both  nms,  and  makes 
the  same  nondeterministic  choices  (if  any)  in  both  runs.  Consider  a  run  r  of  a  system.  If  r  is  not 
in  FS  but  is  isomorphic  with  respect  to  t  to  a  run  r'  in  FS,  then  the  events  t  executes  are  the  same 
as  if  it  were  running  in  a  system  satisfying  the  fidl-stop  assumptions.  Hence,  if  r  =p  r',  then  no 
process  in  P  can  determine  that  r  is  not  in  FS. 

Definition4  A  fiiUure  model  M  is  inddstmgaishahle  from  the  foil-stop  model  if  for  any  run  r  e  M, there 
exists  a  run  r'  €  FS  such  that  =p  r'  (that  is,  r  is  indistinguishable  from  r'  to  every  process  in  P). 

Consider  the  election  protocol  described  in  Section  1.  If  a  nm  of  this  protocol  is  in  a  feilure 
model  M  that  is  indistinguishable  from,  but  not  identical  to  FS,  then  there  may  be  more  than  one 
leader  in  some  global  state,  but  no  process  will  be  able  to  determine  this.  Thus,  internally  the 
execution  is  the  same  as  if  there  were  only  one  leader  at  a  time. 

Recall  that  the  reason  that  FS  can  not  be  implemented  in  an  asynchronous  system  is  because 
the  crash  of  a  process  caimot  be  reliably  detected.  A  failure  model  M  that  can  be  implemented 
and  is  indistinguishable  from  FS  must  be  weaker  than  FS.  However,  it  caimot  be  too  weak;  at  the 
very  least,  a  process  i  must  not  be  able  to  determine  that  some  process  j  executes  an  event  after 
t  detects  that  j  has  crashed.  Furthermore,  if  a  process  detects  the  feilure  of  t  then  t  must  crash 
at  some  point,  and  process  crashes  must  have  been  able  to  occur  in  some  total  order.  Hence,  the 
following  three  conditions  are  necessary  for  indistinguishability  from  FS. 

Condition  1  For  all  runs  r,  i/r  ^  OFAILEDi(j),  then  r  f=  OCRASHj. 
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Condition  2  The  failed-before  relation  must  be  acyclic.  That  is, for  all  runs  rand  for  all  k,  there  cannot  exist 

processes  X\,X2,-..,Xk  such  that  r  FAILED,,  ( I2 )  A  FAILED,,  ( I3 )  A  •  •  •  A  FAILED,,,  _  I  ( t )  A  FAILED,*  ( i  ] ). 

Condition  3  For  all  runs  r,  there  cannot  be  an  event  e  of  process  j  such  that  faHediij )  ~  ein  Hr  ■ 

Theorem  2  If  failure  model  M  is  indistinguishable  from  FS,  then  all  runs  of  M  satisfy  Conditions  1-3. 

Proof: 

Condition  1  In  order  for  two  runs  to  be  isomorphic,  their  histories  must  contain  the  same  events. 
Fbr  every  run  r  that  satisfies  FS,  failed-(j)  €  'Hr  ^crashj  €  Hr-  Therefore,  the  same  must  be 
true  of  every  run  that  satisfies  M .  □ 

Condition  2  For  contradiction,  suppose  that  there  is  some  run  r  of  M  such  that  r  does  not  satisfy 
Condition  2.  We  show  that  there  is  no  run  r'  satisfying  FS  that  is  isomorphic  to  r  with  respect 
to  P. 

If  r  does  not  satisfy  Condition  2,  then  there  is  some  set  of  processes  {xo,xi, . .  .,Xfc}  such  that 
?<,  =  (•••  failedtoixi)  •  •  ’  failed^^(x2)  failed^^_^(xk)  •  •  -fitiled^^^xo)  •  •  •)•  Fo*" nm 

r'  satisfying  FS,  Hr'  must  contain  crashg.  fbr  all  0  <  t  <  A;.  Furthermore,  crash^  must  occur 
before ^iledjp.gj(xj)  and^ilai,.(ii®i)  must  occur  before  crash^i  where  ©  and  ©  are  -  and  + 
modulo  A;  +  1  respectively.  By  transitivity,  this  leads  to  dirular  oonstiaints  on  7ir>:  crashg^ 
must  occur  before  failed^^^xo),  which  must  occur  before  crashg^,  which  must  occur  before 
(**)'  •  •  roust  occur  before /ailcd^^(xi),  which  must  cxxnir  before  crasAix^.  It 

is  impossible  to  satisfy  all  of  these  ordering  constraints  in  a  valid  run.  Therefore,  there  is  no 
run  r'  isomorphic  to  r  that  satisfies  FS.  □ 

Condition  3  Fbr  ccmtradiction,  suppose  that  there  is  some  nm  r  of  M  such  that  r  does  not  satisfy 
Condition  3.  We  will  show  that  there  is  no  run  r'  satisfying  FS  that  is  isomorphic  to  r  with 
respect  to/*. 

If  r  does  not  satisfy  Condition  3,  then  Hr  =  {•••  failed^{j)  •  •  •  sendi{k,  jn^)  •  •  •  recvj{l,  m^)  •  •  • 
ej  •  •  •),  where  sendi{k,  mk)  -*  recvj{t,  mj).  Fbr  any  r'  isomorphic  to  r,  Ht>  must  maintain 
the  order  of  failed^{j),  sendi{k,  mk),  and  recvj((,  mj)  in  order  to  satisfy  the  happens-before 
relation.  However,  for  r'  to  satisfy  FS,  crashj  must  occur  before  failed^(j)  in  Hr'.  This  means 
that  m  Hr',  crashj  must  cxxur  before  recoj(i,  mj),  which  contradicts  the  definition  of  crashj. 
Therefore,  there  is  no  nm  r'  isomorphic  to  r  that  satisfies  FS.  □ 

We  have  shown  that  Conditions  1,  2,  and  3  are  necessary  for  a  failure  model  to  be  indistin¬ 
guishable  from  foil-stop.  However,  these  conditions  are  not  sufficient. 

Theorem  3  There  exists  a  run  r  that  satisfies  Conditions  1-3  such  that  ->3t'  :  r'  =p  r  A  r'  e  FS. 
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Proof;  Let  r  be  the  hawing  run; 

failed^(x);sendy{a,  Tna):recVa(y,  ma): crasha,foiled^{a);sendi,{x,  mx);recvr{b,  mi)-,crasht  ■  •  • 

For  any  r'  isomonphic  to  r,  v  e  have  the  following  ordering  constraints  on  'Ht<' 

•  faded^ix)  -*  sendy(a,ma)  ->■  recDa{y,ma)  —  crasho 

•  fdiled^{a)  —  sendb{x,  nix)  -*  recDx{b,  mx)  -»  crashx 

•  crashx  must  occur  before  fiit7ed^(z) 

•  crasha  must  occur  before a) 

It  is  impossible  to  satisfy  all  of  these  ordering  constraints  in  a  valid  run.  Therefore,  there  is  no 
run  r'  isomorphic  to  r  that  satisfies  FS.  □ 

Theorem  3  imjdies  that  a  failiue  model  M  that  satisfies  Conditions  1-3  may  not  be  indistin¬ 
guishable  from  FS.  In  the  next  section,  we  give  a  set  of  conditions  that  are  sufficient,  thou^  not 
all  are  necessary. 

33  Simulated  Fail-Stop 

We  give  four  properties  that  comprise  a  model  that  is  indistinguishaUe  from  foil-stop.  We  call 
this  model  the  simulated  fdU-stop  model  (sFS). 

To  construct  conditions  for  the  sFS  modd,  we  weaken  one  of  the  conditions  of  the  foil-stop 
modd.  Weakening  FSl  yidds  a  modd  in  which  some  failures  may  be  undetected.  Under  such  a 
modeL  it  could  be  impossible  for  a  system  to  make  progress.  Therefore,  we  follow  [CI91,CHT92, 
RB91]  and  weaken  FS2.  This  yields  a  modd  in  which  nonexistent  foiluies  may  be  detected. 

FSl  is  a  liveness  property.  In  a  real  system,  it  would  be  be  implemented  using  timeouts;  each 
process  would  periodically  send  a  message  to  every  other  process.  If  process  t  were  not  to  recdve 
a  message  from  process  j  within  some  predetermined  length  of  time,  then  t  would  (perhaps 
erroneously)  detect  the  failure  of  j.  We  assume  for  the  remainder  of  this  paper  that  there  is  smne 
mechanism  provided  by  the  tmderlying  system  to  implement  FSl. 

We  replace  FS2  with  the  following  four  conditions; 

8FS2a;  Vr,i,j:  r  (=  □(FAILEDj(j)  =►  ^CRASH,) 

This  condition  states  that  if  process  t  detects  that  process  j  has  crashed,  then  eventually  j  will  crash 
even  if  t's  detection  was  erroneous.  In  conjunction  with  FSl,  this  condition  implies  Condition  1; 
if  fmled-{j)  occurs  in  Hr,  then  crashj  occurs  in  Tfr- 

sFS2b ;  The  foiled-before  relation  is  always  acyclic 
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sFSl:  FSl 

sFS2a:  r  ^  □(FAILEDi(j)  =>  OCRASHj) 

8FS2b:  The  failed-before  relatioiv  is  acyclic. 

sFS2c:  r  |=  □-.FAILEDj(i) 

sFS2d:  r  |=  □[FAILED, (j)  A  ->SEND,( ib,  m)  =>• 

0((SEND.(*,  m)  A  RECV*(t,  m))  =>  FAILED*(i))] 


Hgure  1:  Simulated  Fail-Stop  Conditions 


This  is  Condition  2 

sFS2c:  Vr,  i:  r  [=  □-.FAILEDi(i) 

Hus  condition  states  that  a  process  never  detects  its  own  failure.  That  is,  foiled^{i)  does  not  occur 
in  W,. 

8FS2d  :  Vr, i,jyk:  r  [=  □{FAILED<(i)  A  -iSENDi(ib, m)  =► 

□((SENDi(ib,  m)  A  RECVfc(i,  m))  =»  FAnJED*(i))] 

This  condition  states  that  once  t  detects  the  failure  of  j,  then  any  subsequent  messages  sent  by  t 
to  any  process  k  will  not  be  received  until  k  has  also  detected  the  failure  of  j.  That  is,  if  sendi(k,  m) 
occurs  after /biledj(j)  in  Hr,  thenfdUed^{j)  occurs  before  recvk{i,  m)  in  Wr . 

Properties  sFS2c  and  sFS2d  together  imply  Condition  3,  as  shown  in  the  following  lemma. 

Lemma  4  If  sFS2c  and  sFS2d  hdd  in  a  run  r,  then  there  cannot  be  an  event  e  of  process  j  such  that 
Mledi(j)  -*•  e  in  Hr- 

Proof-.  Consider  any  run  r.  If  t  =  j,  then  the  lemma  is  trivially  true,  because  from  sFS2c,  finled-{i) 
does  not  appear  in  TU-  Assume  that  t  ^  j.  For  contradiction,  let  e  be  an  event  of  j  such  that 
e  in  Hr-  Since  fdiled^(j)  and  e  are  of  different  processes,  from  the  definition  of  the 
happens-before  relation  thereis  a  sequence  of  events^i7ed,(j)  -►  sendi{ki,mkf)  -*  recvjfj(i,mkj)  ->■ 

sendjtj  (*2,  — - -  Tecvj(ku  )-*e.  Rom  sFS2d,  each  pmocess  in  this  chain,  including  j, 

must  have  detected  the  failure  of  j  by  the  time  it  receives  its  message.  Therefore,  failed -{j)  mu^ 
occur  in  Hr,  whidi  contradicts  sFS2c  □ 

The  sFS  conditions  are  summarized  in  Hgure  1. 


Theorem  5  The  simulated  fail-stop  model  (sFS)  is  indistinguishable  from  the  fail-stop  model  (FS). 


The  full  proof  of  this  theorem  is  given  in  Appendix  A.2.  An  outline  of  the  proof  is  given  below. 

Consider  a  run  r  that  satisfies  FSl  and  sFS2a-d  but  violates  FS2.  Then,  there  exists  at  least  one 
pair  of  processes  i  and  j  such  that  r  ^  0(FAiLEDj(i)  A  ->CRASHi).  For  each  such  pair,  by  sFS2a, 
r  OCRASHj.  Therefore,  Kr  =  (•  ■  'failed-{i)  •  •  'crashi  •  •  •).  It  can  be  shown  that  an  event  e  can 
be  moved  within  Hr,  resulting  in  Hri  such  that  r'  =p  r,  as  long  as  the  happens-before  relation  is 
maintained  in  Ht‘-  We  show  in  Ajypendix  A2.  that  -*(finledj(i)  -*  crashi),  and  that  therefore,  crashi 
and  all  events  e  between  failed^{i)  and  crashi  in  Hr  such  that  e  crashi  can  be  moved  to  precede 
failed-{x)  in  Hr>-  Thus,  if  r  satisfies  sFS2a-d,  then  the  events  in  Tfr  can  be  rearranged  so  that  crashi 
precedes for  all  t,  j  in 

4  Lower  Bounds 

The  simulated  fail-stop  properties  (FSl,  sFS2a-d)  put  restrictions  on  the  way  in  whidi  failures  are 
detected.  Implementing  these  properties  requires  that  processes  follow  a  protocol  for  detecting 
failures.  In  this  section,  we  give  lower  bounds  on  message  complexity  and  replication  for  failure 
detection  protocols  implementing  sFS. 

Aone-round  protocol  for  detectings  failure  is  one  in  which  eadi  process  t  exchanges  one  round  of 
messages  with  other  processes  before  executing Any  protocol  simpler  than  a  one-round 
protocol  would  allow  at  least  one  process  to  unilaterally  detect  the  failure  of  some  other  process. 
Such  a  protocol,  however,  would  limit  which  processes  another  process  could  detei.;  as  faulty. 
For  example,  suppose  that  process  t  can  unilateraUy  decide  that  pnxress  j  has  failed.  Process  t 
can  execute  faUai-(j)  concurrently  with  any  event  of  process  j,  and  so  process  j  can  never  execute 
fdiled^{i).  Hence,  we  will  consider  the  class  of  one-round  protocok  in  order  to  determine  message 
and  replication  complexity. 

We  say  that  a  process  t  initiates  a  failiue  detection  protocol  when  it  "suspects"  the  failure 
of  another  process  j  (e.g.,  due  to  a  timeout  at  a  lower  level)-  In  the  first  half  of  the  round, 
process  i  sends  a  message  to  all  other  processes;  in  the  second  half  of  the  roimd,  processes  send 
an  acknowledgement  message  to  i.  We  call  the  first  message  SUSPij  and  the  acknowledgement 
message  ACKSUSPi^ .  Upon  completion  of  the  failure  detection  protocol,  i  will  execute  either  crashi 
or  for  some  j  ^  i. 

A  one-round  protocol  that  implements  sFS  must  avoid  cydes  in  the  fiuled-before  relation  since 
all  runs  in  sFS  satisfy  sFS2b.  Implementing  8FS2b  requires  that  in  any  run  there  is  at  least  one 
process  that  partidpates  in  all  foilure  detections.  To  see  why  this  is  so,  ccmsider  tiie  problem  of 
avoiding  cycles  involving  exactly  two  processes.  Suppose  that  process  a  suspects  the  failure  of 
process  b.  Before  a  can  execute  fmled^(b),  the  failure  detection  protocol  must  ensure  that/!n7ed{,(a) 
has  not  been  executed  and  that/!n7a/(,(<i)  will  not  be  executed  in  the  future. 

Ihe  failure  detection  protocol  cannot  require  a  to  conununicate  with  b  directly,  because  b  may 
have  indeed  crashed.  Therefore,  the  protocol  must  require  a  to  receive  mformation  from,  and 


distribute  information  to,  other  processes.  In  particular,  a  must  receive  information  from  enough 
other  processes  to  be  sure  that^i7c<<(,(a)  has  not  been  executed,  and  a  must  distribute  information 
to  enough  other  processes  to  be  sure  that  if  failed^  (b)is  executed,  then  failed  ( a )  wiU  not  be  executed 
in  the  future. 

The  relevant  information  that  a  must  disseminate  is  that  a  suspects  the  faUxire  of  6.  In  order 
for  a  to  know  that  this  information  has  been  received  by  other  processes,  it  must  receive  messages 
from  other  processes  acknowledging  that  the  failure  of  6  is  suspected. 

Definition  5  The  quorum  set  Qij  a/feiled,(j)  is  the  set  of  processes  from  which  i  has  received  acknowl¬ 
edgement  messages  relating  to  its  suspicion  of  j‘s  crash.  Formally,  Qij  =  {k  e  P  :  SEND, (h,  SUSP, j)  A 
RECVi(fc,  ACK^USPij)}. 

The  set  Qab  must  be  large  enough  to  ensure  that  b,  after  hearing  hrom  Qta,  will  not  execute 
failed^(a).  In  particular,  the  sets  Qab  ^60  must  have  a  non-null  intersection. 

We  call  this  property  the  Witness  Property  (W),  because  the  quorum  sets  for  any  two  failure 
detections  must  have  at  least  one  process  (the  witness)  in  common.  It  can  be  shown  that  the  same 
property  must  hold  in  order  to  avoid  cycles  of  any  size.  The  Witness  Property  can  be  stated 
formally  as  follows: 

m  n  Qa  ^  0 

Vt  J  FAILED.(i) 

That  is,  there  is  some  process  w  that  is  in  the  quorum  set  of  all  failure  detections.  Note 
that  this  is  a  stronger  condition  than  what  is  necessary,  for  example,  in  the  update  of  replicated 
variables  [Gif79]  in  which  only  each  pair  of  quorum  sets  must  intersect. 

Theorems  (Vr:  r  |=  □8FS2b)  =»  (Vr:  r  \=  nW). 

It  was  argued  above  that  (r  |=  OsFSZb)  =»  (r  f=  DW)  if  only  cydes  of  size  two  are  possible. 
The  full  proof  of  the  theorem  is  given  in  Appendix  A.3. 

Since  sFS2b  (Condition  2)  is  necessary  for  indistinguidiability  ftom  FS  (see  Section  3.2),  The¬ 
orem  6  implies  that  >V  is  necessary  for  any  one-round  protocol  that  implements  a  failure  model 
indistinguishable  from  FS.  Let  t  be  the  maximum  number  of  crashes  in  any  run,  induding  those 
that  arise  from  erroneous  suspidons.  The  necessity  of  the  V^^tness  Property  places  a  constraint  on 
t  as  a  function  of  n  and  on  the  number  of  messages  that  a  process  must  wait  for  beftne  detecting 
afeilure. 

The  simplest  way  to  ensure  that  W  holds  in  a  one-roimd  protocol  is  to  require  a  process  to  wait 
for  responses  from  every  other  process,  except  for  those  that  are  suspected  to  have  foiled,  before 
detecting  a  foilure.  If  there  is  always  at  least  one  process  that  never  foils,  nor  is  suspected  of  failing, 
then  this  process  will  be  a  witness  to  every  failure  detection  that  is  executed.  This  implementation 
(mly  requires  that  t  <  n.  However,  if  n  is  large  and  t  is  small,  then  each  foilure  detection  requires 
a  process  to  wait  for  many  messages,  which  in  practice  could  take  a  long  time. 
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An  alternative  implementation  is  to  require  a  process  to  wait  for  a  fixed,  predetermined  number 
of  responses  before  detecting  a  failure.  This  approach  reduces  the  size  of  the  quorum  for  which  a 
process  must  wait,  but  it  places  a  stronger  restriction  on  the  number  of  failures  that  can  occur. 

Theorem  7  Ifthesizeofthetfuorum  set  isa  fixed  and  equal  size  for  each  failure  detection,  then  to  guarantee 
that  r  f=  □  W  when  t  failures  are  possible,  the  size  of  each  quorum  set  must  be  strictly  greater  than  n(  ^). 

Proof:  We  assume  that  in  any  run,  no  more  than!  failures  will  occur.  Therefore,  the  largest  possible 

cycle  in  a  run  satisfying  (simulated)  fail-stop  involves  t  processes.  We  must  guarantee  that  any  t 

quorum  sets  have  a  nonempty  intersection. 

Let  the  size  of  a  quorum  be  x.  Let  y  =  n  -  i.  Suppose  y  =  [7].  Then  there  is  a  set 

of  t  quorum  sets  such  that  Vi  e  P  :  3j  :  *  ^  Qj-  In  particular,  let  Qy  =  P  -  {1,2, ...,y}, 

Q2  =  P-{y+l,y  +  2,...,2y},  •  •,  gt  =  P-{n-y  +  l,n-y-|-2,...,n}.  By  construction, 

( 

each  process  is  not  a  member  of  at  least  one  quorum.  Therefore,  P|  g,-  =  0.  Qearly,  such  a  set  of 

t=i 

quorum  sets  can  also  be  constructed  if  y  >  ffl .  Therefore,  we  must  have  y  <  f  . 


j  =  n-  y  =>•  x>n-fj') 

,nt-n 

=►  *  >  I- ---J 

Th  erefore,  the  size  of  a  quorum  must  be  an  integer  strictly  greater  than  w(  ^).  □ 

CoroUaiy  8  If  the  minimum  quorum  size  is  used  in  a  one-round  protocol  for  failure  detection,  then  it  must 
be  the  case  that  n  >  t^. 


Proof:  in  a  one-round  protocol,  the  size  of  the  quorum  is  equal  to  the  number  of  aCK.SUSP,-j 
messages  that  process  i  must  receive  before  executing  failed^{j).  Since  t  is  in  its  own  quorum,  i 
must  wait  for  [n(  messages  before  detecting  j's  failure.  In  order  for  the  one-round  protocol 

to  make  progress,  at  least  this  many  other  processes  must  remain  alive.  Therefore,  we  have 

n-t>  Ln(^-j^)J  =►  n-t>  [n-j\>n  - 

=>  ^<17] 

=>  t^  <n 


□ 
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5  Upper  Bounds 

We  give  a  simple  one-rotind  protocol  that  implements  sFS2a-d.  We  assume  that  a  failure  can  be 

suspected  spontaneously  (e.g.,  due  to  a  timeout),  but  that  no  more  than  t  failures  are  suspected  in 

any  run.  In  this  protocol,  SUSP,,^  =  ack.SUSP,  j  =  “j  failed". 

•  When  process  i  suspects  the  failure  of  process  j,  i  sends  the  message  "j  failed"  to  all  processes 
(including  itself).  Process  i  waits  for  messages  of  the  form  "7  failed"  from  other  processes 
and  takes  no  other  action  except  for  acknowledging  "x  failed"  messages  until  it  completes 
the  protocol  or  crashes. 

•  When  process  i  has  received  messages  of  the  form  "j  failed"  horn  more  than  n(  ^ )  processes 
(including  itself),  i  executes  foiled -(j). 

•  When  process  x  receives  a  message  of  the  form  "x  failed",  x  executes  crash,;. 

•  When  process  x  receives  a  message  of  the  form  "y  failed",  x  suspects  the  failure  of  y. 

We  will  argue  informally  that  this  protocol  implements  the  simulated  fail-stop  properties. 

sFS2a:  Process  t  cannot  execute  foiled -{j)  without  sending  a  message  of  the  form  "7  failed"  to  all 
other  processes,  including  7.  Since  channels  are  nonfaulty,  7  will  eventually  receive  such  a 
message,  upon  which  7  will  crash. 

sFS2b:  The  full  proof  is  given  in  Appendix  A.4.  We  give  an  outline  of  the  proof  for  cydes  of  length 
2.  Suppose  that  the  protocol  generates  a  run  r  such  that  r  ^  0(FAILEDj(7)  A  FAILEDj(t)).  By 
Theorem  7,  r  □  W  holds.  Therefore,  there  is  some  witness  w  such  that  i  received  "7  failed" 

from  w  and  7  received  "t  failed"  from  w.  Process  w  sends  these  messages  to  all  processes.  If 
w  sends  "7  failed"  before  it  sends  "t  failed",  then  process  7  will  receive  "7  failed"  and  crash 
before  it  can  execute  foil^j{i).  Similarly,  if  w  sends  "t  failed"  before  it  sends  "7  failed",  then 
process  i  will  receive  "t  failed"  and  crash  before  it  can  execute /ailed, (7).  Therefore,  it  is  not 
possible  for  both /failed^  (7)  emd  foiled^{i)  to  be  executed  in  a  run. 

sFS2c  Process  i  cannot  execute  foiled-{i)  without  receiving  at  least  one  message  of  the  form  "t 
failed".  Upon  receiving  such  a  message,  i  crashes.  Therefore,  foiled^{i)  is  never  executed. 

8F52d:  Since  channels  are  FIFO,  any  message  m  sent  by  i  to  k  ahet  foiled-{j)  is  executed  must  be 
received  after  the  message  "7  failed".  Upon  receiving  "7  failed"  from  i,  process  k  suspects 
the  failure  of  7  and  initiates  the  failure  detection  protocol.  Process  k  does  not  receive  m 
until  either  cros/ifc  or  foiled^{j)  is  executed.  Hierefore,  message  m  is  not  received  by  k  unless 
foiled i^{j)  has  been  executed. 
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6  Discussion 


In  Section  3.2,  we  showed  that  Conditions  1, 2,  and  3  are  necessary  for  any  failure  model  to  be 
indistinguishable  from  the  fail-stop  model.  In  Section  4,  we  showed  that  the  Witness  Property 
is  necessary  for  any  one-round  protocol  implementing  Condition  2.  We  then  showed  that  the 
Wtness  Property  imposes  lower  bounds  on  the  number  of  messages  that  must  be  received  before 
a  feiilure  can  be  detected  and  on  the  number  of  hiilures  that  can  be  tolerated  in  a  system. 

We  gave  a  protocol  in  Section  5  to  demonstrate  that  these  bounds  are  tight.  This  protocol, 
however,  was  derived  from  conditions  that  are  not  necessary  for  indistinguishability.  There  may 
be  a  ^ure  model  weaker  than  sFS  that  is  indistinguishable  from  FS.  However,  such  a  ^ure 
model  is  subject  to  the  same  bounds  on  t  as  sFS,  and  so  we  do  not  expect  such  a  failure  model  to 
be  substantially  more  interesting  than  sFS. 

The  boimds  on  ( arise  from  sFS2b.  A  ^ure  model  satisfying  only  the  other  sFS  assumptions 
would  not  require  a  process  to  wait  for  any  messages  before  detecting  a  tidlure:  the  other  sFS 
properties  can  be  implemented  simply  by  having  process  t  broadcast  a  message  "j  failed"  after 
suspecting  j's  ^ure  and  before  unilaterally  executing ^t7ed,(j).  Such  a  tiulure  model  would,  of 
course,  be  distinguishable  from  FS,  but  if  a  collection  of  processes  are  insensitive  to  cyclic  hulures, 
then  they  could  be  run  in  this  cheaper  simulated  failure  model.  We  do  not  know  of  any  protocols 
in  the  literature  that  are  insensitive  to  cyclic  failure  detection,  however. 

As  an  example  of  sensitivity  to  sFS2b,  consider  the  problem  of  determining  the  last  process  to 
^  ([Ske85]).  Solving  this  problem  requires  that  processes  record  information  about  the  failiues 
that  they  detect  (that  is,  their  view  of  the  failed-before  relation).  Then,  when  processes  are 
recovering  after  a  total  failure,  the  recovering  processes  can  determine  when  the  last  processes  to 
hul  have  recovered.  If  cyclic  failme  detection  is  possible,  then  the  problem  is  not  solvable.  For 
example,  suppose  P  =  {1, 2},  process  1  hdsely  detects  2's  failure,  and  then  crashes.  Process  2 
detects  I's  failure,  proceeds  with  its  work,  and  finally  crashes.  If  process  1  were  to  then  recover,  it 
would  conclude  that  it  was  the  last  to  fidl.  In  general,  if  cychc  detection  is  possible  then  the  only 
possible  recovery  is  to  always  wait  for  all  crashed  processes  to  recover. 

There  are  other  protocols  that  require  failure  models  even  stronger  than  FS.  For  example,  if 
the  failed-before  relation  is  transitive  as  well  as  acychCr  then  detecting  the  last  process  to  fail  can 
be  implemented  so  that  as  soon  as  the  last  processes  to  fail  have  recovered,  then  the  processes  can 
determine  this.  If  the  fiuled-before  relation  is  not  transitive,  then  it  is  necessary  to  wait  for  more 
processes  to  recover.  The  ^ed-before  relation  of  sFS  is  not  transitive.  We  are  currently  looking 
into  several  stronger  versions  of  foil-stop,  whether  they  are  implementable  given  foil-stop,  and 
into  how  they  too  can  be  simulated. 

The  protocols  described  in  this  paper  are  very  simple  and  are  easily  implementable.  Failure 
detection  sudi  as  described  here  is  typically  done  as  part  of  a  group  member^p  service  (eg., 
[RB91,MPS91,ADKM92]).  We  believe  that  the  protocols  here  could  be  used  as  the  basis  of  a  failure 


14 


detector  that  could  be  used  outside  of  a  system  built  using  a  group-membership  protocol.  This 

would  aUow  for  consistent  failure  detection  on  top  of  any  kind  of  lower-level  communication, 

including  point-to-point  communication. 
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A  Appendices 

A.1  Formal  Definition  of  Events,  Runs,  and  Special  Predicates 

Recall  that  an  event  e  is  a  function  that  maps  global  states  to  global  states.  An  event  e  applied  to  a 
global  state  S  either 

•  yields  E,  in  which  case  we  say  that  e  is  a  null  event;  or 

•  yields  a  new  global  state  E'  that  differs  from  E  in  the  local  state  of  exactly  one  process  i  and 
the  state  of  at  most  one  channel  incident  on  i.  We  say  in  this  case  that  e  is  an  event  of  i,  and 
that  e  changes  the  state  of  i. 

A  non-null  event  e  is  imiquely  defined  by  the  process  i  whose  state  it  changes,  the  state  s  of  i 
immediately  before  e  is  applied,  the  state  s'  of  t  resulting  from  e,  the  states  of  the  chaimels  incident 
on  i  before  e  is  applied,  and  the  states  of  the  chaimels  incident  on  i  after  e  is  applied.  Let  A',  j  be 
the  state  of  channel  Let  A'i,,  be  the  n-tuple  Xi^n)  and  let  X,^i  be  the  n-tuple 

(A'l,.,  ^2,1,  • .  • ,  A:’n,.)  Then,  e  is  defined  by  the  7-tuple  {i,  s,  s',  A’j,,,  X'i  ,,  X„^i,  such  that: 

•  if  A'i,,  ^  X'i  „  (e  is  a  send  event),  then  A',,,  =  X'„  i,  there  exists  exactly  one  j  ^  i  such  that 
Xij  ^  X'i  j,  and  X'i^  =  ( A",  j  ::  m)  for  some  message  m  (where ::  is  the  catenation  operator). 

•  if  A^,,<  X'^  i  (c  is  a  receive  event),  then  X,^,  =  A'J  ,,  there  exists  exactly  one  j  ^  i  such  that 

Xj^i  X'j  i,  and  (m ::  X'^^)  =  A'j.i  for  some  message  m. 

If  e  is  a  null  event,  then  e  is  not  of  any  process  t  and  therefore  is  not  represented  by  a  7-tuple. 

Definition  6  Wc  soy  that  (non-null)  e  =  {i,  s,  s',  A',-,.,  A’J  A:',,^,  A", ,)  can  ocean  in  global  state  E  if  and 
only  if: 

•  the  state  of  process  tin  D  is  s, 

•  the  states  of  the  incoming  channels  incident  oniinE  are  A',,.,  and 

•  the  states  of  the  outgoing  channels  incident  on  i  in  E  are  X,,,. 

A  mdl  event  can  occur  in  arty  state. 

Let  e  =  {i,  s,  s',  A'i,*,  X'i  ,,  A'.,„  At,,).  We  abbreviate  send  and  receive  events  as  follows. 

•  If  e  is  a  send  event  of  i  and  =  (Cij  ::  m)  for  some  j,  then  e  is  denoted  sendi{j,  m). 

•  If  e  is  a  receive  event  of  i  and  (m ::  ,)  =  Cj,i  for  some  j,  then  e  is  denoted  recvi(j,  m). 

We  define  "crash"  events  and  "failure  detection"  events  as  follows: 
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•  If  crashi  is  false  in  s  and  true  in  s',  then  e  is  denoted  crash,.  By  assumption,  crash,  changes 
only  the  local  variable  crash,  . 

•  If  3j :  failed,  (j)  is  false  in  s  and  true  in  s',  then  e  is  denoted  failed -{j). 

The  events  sendi{j,  m),  recvi{j,  m),  crashi,  and  failed-{j )  are  atomic;  each  event  only  changes  the 
relevant  state  variables  of  the  process  on  which  it  occurs.  For  example,  if  crash^  is  false  in  local 
state  5  of  t  when  send,(j,  m)  occurs,  then  crash,  is  false  in  the  resulting  state  of  t. 

Definition  7  Let  r  =  (So,  Si,  Sa,  ...)bean  infinite  sequence  of  global  states  of  the  system.  V\fe  say  that 
r  is  a  run  of  the  system  if  and  only  if  So  is  an  initial  global  state  and  there  exists  a  sequence  of  events 
(cfl,  Cl,  C2, . . .)  such  that  for  all  i  >  0,  c,  can  occur  in  S,  and  Sj+i  =  e,(Sj). 

•  SENDi(j,  m)  and  RECVi(j,  m)  are  Mse  in  an  initial  global  state. 

•  Lete  =  scmij(j,  m)  and  lets  be  a  global  state  such  that  e  can  occur  in  S.  Thensen(f,(j,m)(S)  h 
SEND<(j,  m).  That  is,  SENDi(j,  m)  becomes  true  when  sendi{j,  m)  has  occurred. 

•  Lete  =  recn{(j,m)andletSbeag;lobalstatesudithatecanoocurinS.  Thenrecv,(7, m)(S)  \= 
RECV,(  j,  m).  That  is,  RECVj(j,  m)  becomes  true  when  recvi{j,  m)  has  occurred. 

A.J2  Proof  of  Theorem  5 

Theorem  5  The  simulated  fail-stap  model  fsFSi  is  indistinguishable  from  the  fail-stop  model  CES). 

hi  order  to  prove  that  for  any  run  r  that  satisfies  FSl  and  sFSZaHl,  there  is  an  isomorphic  run 
r'  that  satisfies  FSl  and  FS2,  we  will  need  to  determine  the  conditions  under  which  an  event  in  a 
history  "Hr  can  be  moved  to  yield  a  history  Hr'  such  that  r  =p  r'. 

Consider  Hr  =  (••.€»,  e,+i ,  e,+2  •  •  •)  corresponding  to  run  r  =  (. . . ,  S^,  S,+i,  Si+2,  • .  •)•  ®y 
definition,  e,-  can  occur  in  S,  and  e,>i  can  ocrur  in  e,(S{)  =  S^^.!.  Assume  that  e,-  and  6^4.1  are 
ncm-nuU  events. 

Suppose  that  e,-  and  £,>1  are  of  the  same  process  k.  Since  e,-  changes  the  state  of  k,  the  state  of 
k  is  not  the  same  in  £<  as  in  £{4.1.  Therefore,  £,4.1  cannot  occur  in  S,-. 

Now  suppose  that  £,-  and  £,-4.1  are  of  two  different  processes  k  and  t,  respectively.  The  state  of 
f  in  S,  is  the  same  as  that  in  £{4.1,  because  £,  does  not  change  the  state  off.  Therefore,  if  £,-4.1  is  not 
a  receive  event,  then  £,4.1  can  occur  in  S,  .  If  £<4.1  is  a  receive  event,  and  changes  the  state  of  any 
incoming  dianndotherthanCjb^,  then  £{4.1  canoccurin  E„  because  the  states  of  all  other  incoming 
channels  must  be  the  same  in  S,-  and  £{4.1.  However,  if  £^4.1  =  recvt{kj  m)  and  £,-  =  sendk{l,  m), 
then  £,-4.1  cannot  occur  in  S,-,  because  the  message  m  is  not  part  of  in  S,  . 

In  summary,  £,-4.1  cannot  occur  in  S,-  if  and  only  if 

•  £{  and  £{4.1  are  of  the  same  process,  or 
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•  e,-  =  sendk{i,m)  and  ei+\  =  recvt{k,m). 

In  other  words,  e^+i  cannot  occur  in  S,  if  and  only  if  (e^  —  e,+i ). 

Assume  that  can  occ  -  in  E„  and  let  =  e,+i(E,).  It  can  be  shown  by  a  similar 
argument  that  cannot  change  the  state  of  k,  or  A'f  jc  in  such  a  way  as  to  violate  the 
preconditions  for  Cj,  so  e,  can  always  occur  in  Furthermore,  e,(ei+i(Ei))  =  ei+i(ei(Ei)). 
Therefore,  r'  =  (...E,,E:^,  ,  Ei+2>  • .  •)  is  a  valid  run,  where  Tir’  =  (■■■  e,+i ,  e„  e,+2  •  •  •)• 

Consider  t)  •  (Recall  that  repeated  states  are  removed  in  these  sequences.)  From 

the  construction  of  r',  r/c  =  r^.  and  r/  =  r'f.  Since  e,  and  e,>]  do  not  change  the  states  of  processes 
other  than  k  and  t,  =  r{  for  all  process  t  ^  {!:,  1).  Hierefore,  r  =/>  r'. 

In  slunmary,  we  have  shown  that  if  ->{€;  ej^.i)  in  Hr,  then  e^+i  can  be  moved  before  e,  to 
.yield  Hr'  such  that  r'  =p  r.  It  can  also  be  shown  that  for  any  two  events  and  ej  in  Kr  such 
that  i  <  j  and  ->(«<  Cj),  ej  can  occur  in  Ej,  ej  can  occur  in  ej(Ej),  and  ej(ej(Ej))  =  Cj(ej(Ej)). 
Therefore,  tj  can  be  moved  to  directly  before  Cj  to  yield  Wr'  such  that  r  =p  r'. 

We  can  now  prove  the  theorem. 

Proof:  If  run  r  satisfies  FS2  then  the  theorem  trivially  holds,  so  we  assume  that  r  violates  FS2. 
Then,  there  exists  at  least  one  pair  of  processes  i  and  j  such  that  r  {=  0(FAiLEDj(i)  A -<a?ASHj).  For 
each  such  pair,  by  sFS2a,  r  ^  OCRASHj.  Therefore,  W,  is  of  the  form  (•  •  ’  failed -(i)  •  •  •  crashi  •  •  •). 

Definition  8  A  pair  of  processes  (i,j)  is  bad  in  Hr  if  Hr  —  (•  •  •failedj(i)  •  •  'Crashj  •  •  •).  Otherwise, 
(i,  j)  is  gpod  in  Hr. 

We  prove  the  theorem  by  induction  on  the  number  of  bad  process  pairs  in  Hr- 

Base  case  Assume  that  there  is  only  one  bad  pair  in  Hr-  Let  Hr  =  {x;  failed j{i);  y;  crashi;  where 
X,  y,  and  z  are  sequences  of  events.  Let  k  be  the  number  of  events  in  y.  We  construct  by  induction 
on  A;  a  run  r' isomorphic  tor  such  that  7ir'  =  ix';crashi;piiledj{i);y';z)vfherex'  arid^  areseqaences 
of  events. 

Base  Case  (Inner  Induction)  Assume  A;  =  0.  Hr  =  {x; failed ■{%); crashi;  z).  Since 
crashi  and  fdiled^{j)  are  of  different  processes,  they  can  be  swapped  to  yield  Hr>  = 

(x; crashi;  failed -(iYfZ)  such  that  r'  =p  r.  Clearly,  r'  satisfies  FS2. 

Induction  case  (Inner  Induction)  Assume  that  the  theorem  holds  for  all  histories 
in  which  A:  <  /  -  1,  and  assume  that  k  =  t.  Hr  =  (*;  failedj{i);  e\;  62;  •  •  •;  c/;  crashi; 
z).  By  Lemma  4  we  know  that  ->{failedj{i)  -»•  crashi).  Let  e„  be  the  first  event  of 
(ei;  •  •  • ;  crashi)  such  that  -i(failedj(i)  -»  e«).  Since  e„  is  the  first  such  event  and  -♦  is 
transitive,  Vi :  1  <  x  <  «  :  -i(ei  -►  c„).  IMQ  c  be  the  set  of  processes  sudi  that 
failed j{i),  ei, . . . ,  eu>i  are  events  of  processes  in  Q.  Then  eu  is  an  event  of  a  process  in  Q. 
Therefore,  there  is  a  history  Hr»  =  (x;  failed j{i);  e\;  C2;  •  •  • ;  Cu-i;  Cu+i ;  ■  ■  * ;  e;;  crashi ) 
sudi  diat  r"  =p  r.  By  the  induction  hypothecs  there  is  a  history  Hr'  of  the  desired 
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form  such  that  t'  =p  r",  and  hence  r'  =p  r. 


^ Inner  Induction 


Induction  case  Assume  that  there  are  k  bad  pairs  in  Hr,  one  of  which  is  (i,  y).  We  will  show 
that  we  can  use  the  same  inductive  construction  presented  in  the  Base  Case  to  )deld  a  history  Hr', 
such  that  r'  =p  r,  with  strictly  fewer  bad  pairs,  so  that  the  Inductive  Hypothesis  applies  to  K,-. 

Overview:  Given  a  bad  pair  (i,  y),  consider  another  pair  of  processes  {a,b).  Using  a  case 
analysis  on  all  possible  placements  of  failedhia)  and  crosha  in  Hr  with  respect  to  f(nled^{x)  and 
crash^,  we  show  that  using  the  earlier  inductive  construction,  we  can  "fix"  (i,  y)  —  i.e.,  construct 
a  history  Hr>  in  which  (i,  y)  is  good  —  such  that 

•  if  (a,  b)  is  bad  in  Hr.  then  (a,  b)  is  either  good  or  bad  in  Hp  ', 

•  if  (a,  b)  is  good  in  Hr,  then  (a,  b)  is  either  still  good  in  Ht>,  or  is  bad  in  Hr'  but  can  be 
fixed  without  making  {x,  y)  bad  again  by  using  a  finite  number  of  applicatioits  of  the  same 
inductive  construction. 

There  are  twelve  possible  placements  of  fdiled^{a)  and  crosha  with  respect  to  fdiled^{x)  and 
crashx.  In  each  case,  we  consider  the  efiect  on  (a,  5)  of  applying  the  inductive  construction  to 
ix,y). 

1.  •••  crosha  "•  faUed^{a)  •••  faUed^{x)  •••  croshx 

2.  •  •  •  fmledfi{a)  •  •  •  crosha  •  •  •  failed^{x)  •  •  •  croshx  •  •  • 

3.  •••  failed^(x)  •••  croshx  cro^  •••  ••• 

4.  failed^(x)  croshx  •••  faned^{a)  •••  crosha  ••• 

5.  •••  failed^ia)  •••  failed^(x)  •••  croshx  crosha 

6.  •••  crosha  •••  foiled^ix)  •**  croshx  faOed^^a) 

Since  only  events  that  occur  between  fdiled^{x)  and  croshx  are  moved,  (a,  6)  is  independent 
of  (x,  y)  in  these  sbc  cases,  in  that  fixing  (z,  y)  has  no  effect  on  the  goodness  of  (a,  b).  Thus, 
(z,  y)  becomes  good  and  (a,  b)  is  unchanged. 

7.  •••  fiuJed^(a)  •••  foiledy{x)  •••  crosha  •••  croshx  ••• 

In  this  the  history  Hp  resulting  from  an  application  of  the  construction  of  the  base  case 

has  one  of  two  forms,  depending  on  whether  or  not  foiled^{x)  — *■  crosha’. 

•  Hr'  =  (•  •  •failed^ia)  •  ■ -crosha  -  -  •  croshx', faiied^{x)--  •) 
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•  Hr>  =  (•  •  -faiied^ia)  ■  •  •  crash^;  failed x) -  •  -crasha  ■  ■  •) 

In  either  case,  (x,  y)  is  now  good  and  (a,  6)  remains  bad. 

8.  •  •  •  failed^{x)  ■  ■  •  crasha  •  •  •  crashx  •  -  •  /fliledj(a)  •  ■  • 

In  this  case,  the  history  Hr>  resulting  from  an  application  of  the  construction  of  the  base  case 
has  one  of  two  forms: 

•  Hr'  =  {••  crasha  •  ■  ■crashi;failed^(x)-  •  •failed,^{a)  •  •  •) 

•  Hr'  =  (■  •  •  crashx}  failed^(x)  •  •  -crasha  •  •  -failed^fa)  -  -  •) 

In  either  case,  (x,  y)  is  now  good  and  (a,  b)  i^mains  good. 

9.  •••  crasha  •••  fdiled^(x)  •••  fatled^fa)  •••  crashx  ••• 

In  this  case,  the  history  Hr>  resulting  from  an  application  of  the  construction  of  the  base  case 
has  one  of  two  forms: 

•  Wr'  =  (•  '-crasha  •  •  -failedf^fa)  -  -  -  crashx}  fiiUed^jx)  •  •  •) 

•  Hr'  =  (•  •  -crasha  -  -  -  crashx,  fiiiled^fx)  ■  --faUed^fa)  -  •  •) 

In  either  case,  (x,  y)  is  now  good  and  (a,  6)  remains  good. 

10.  •  •  •  failedji(x)  -  -  -  failedi,{a)  -  •  -  crashx  -  -  •  crasha  -  -  - 

In  this  case,  the  history  Hr'  resulting  from  an  application  of  the  construction  of  the  base  case 
has  one  of  two  forms: 

•  Hr'  =  (--  -failed^fa)  -  -  -  crashx}  fitSed^(x)  -  -  -  crasha  -  -  -) 

•  Hr'  =  (•  •  -  crashx}  fttiled^jx)  -  •  -fitiled^{a)  -  - -crasha  -  -  -) 

In  either  case,  (z,  y)  is  now  good  and  (o,  6)  remains  bad. 

11.  •••  fdiled^(x)  failedhfa)  ---  crasha  crashx  --- 

In  this  case,  the  history  Hr'  resulting  from  an  application  of  the  construction  of  the  base  case 
has  (me  of  four  forms: 

•  Hr'  =  (- --failed^fa)  -  -  -crasha  -  •  -  crashx’,  faded^(x)  -  -  •) 

•  Hr'  —  (•  •  -fidledf^fa)  -  -  -  crashx’,  fiiiled^(x)  -  -  -  crashg  -  •  •) 

•  Hr'  =  (•  ■  -  crashx’,  fidled^jx)  -  - -fttiled^{a)  •  •  -crasha  -  •  -) 

•  Hr'  =  (•  •  -crasha  -  -  -crashx’,iailed^{x)  -  - -fitiled^{a)  -  -  •) 

In  the  first  three  cases,  (z,  y)  is  now  good  and  (a,  b)  remains  bad;  in  the  fourth  case,  (z,  y)  is 
now  good  and  (a,  b)  is  now  good,  thus  reducing  the  number  of  bad  pairs  by  two. 
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IZ  •••  fitiled^{x)  crasha  •••  failed^(a)  •  ■  •  crashx  ••• 

In  this  case,  the  history  Hr'  resulting  from  an  application  of  the  construction  of  the  base  case 
has  one  of  four  forms: 

•  'Hr'  =  (•  •  crttshx;fttiled^(x)  •  •  crosha  ■  •  -failed^ia)  ■  ■  •) 

•  'Hr'  =  (•  •  crasha  •  •  •/ai7edj(a)  •  ••cra^;fdiled^{x)  ■  •  •) 

•  Hr'  =  (•  •  crashg  •  •  -crashxifiiiled^ix)'  •  -fiiiled^ia)  •  •  •) 

•  Hr'  =  (•  •  •failed^{a)  •  ■  •crashx}  faUed^jx)  crashg  ■  •  ■) 

In  the  first  three  cases,  {x,  y)  is  now  good  and  (a,  6)  remains  good.  However,  in  the  fourth 
case,  (z,  y)  is  now  good,  but  (a,  b)  is  now  bad.  Thus,  the  number  of  bad  pairs  may  not  be 
reduced.  Furthermore,  for  each  pair  (i,j)  such  that  jailed-(i)  and  crashi  appear  in  Tf,.  in  the 
same  order  with  respect  to  fiided^(x)  and  crashg  as  failed^(a)  and  crashg,  there  can  be  one  more 
bad  pair  in  Hr>  than  there  is  in 

However,  we  can  construct  a  history  Tf,"  fiom  Hr'  in  the  same  marmer  in  which  Hr'  was 
corutructedfromHr,suchthat(a,6)isgoodin7fr«  arul(z,y)remain8goodin7fr»  as  follows. 

We  have  Hr'  =  (•  •  'failed^(a)  •  •  •cra^;failed^(x)  •  •  •  crashg  •  •  •).  Recall  that  in  the  construc¬ 
tion  of  Hr'  from  Hr,  an  event  e  between  crashg  and  failed^{x)  was  moved  if  and  only  if 
~i(failed^{x)  e).  Thereftxe,since/iiiled^(a)  was  moved  in  the  construction  of  andcnisha 
was  not,  it  must  be  the  case  that  in  both  Tir  andTir' 

■^ifailed^ix)  faUedf,{a))  A  (fmled^{x)  -*■  crashg)  (1) 

As  shown  in  the  case  analysis,  there  are  four  possible  results  of  applying  the  inductive 
construction  to  Hr'-  Either  of  the  first  three  possibilities  yields  a  history  Hr"  in  which  (a,  b) 
is  good  and  (z,  y)  remains  good.  We  daim  that  the  fourth  possibility  cannot  occur. 

Proof:  Suppose,  for  contradktion,  that  Hr"  =  (*  *  •  •  •  crashg; faiJ^^{a)  •  •  -crashg  ■  •  •). 

Then  by  the  eaiiier  argument  it  must  be  the  case  that  in  andTfr" 

-(faUed^ia)  -*  failed^(x))  A  {Jailedtia)  crashg)  G) 

[faSed^{x)  -*  crashg)  in  Hr'  imfdies  that^7ed,(z)  occurs  in  Hr'  by  sFS2d  and  the  definition 
of  happens-before.  Similarly,  (fittled^{a)  crashg)  implies  that  faUed^{a)  occurs  in  Hr'. 
Thus,  Equations  1  and  2  imply  that  in  Hr'  both  failed  J^x)  and  failed  J^a)  occur  in  Hr',  which 
ccmtradicts  sFS2b.  Therefore,  Hr»  carmot  have  the  assumed  form,  so  both  (a,  b)  and  (z,  y) 
must  be  good  in  Hr". 

Thus,  if  fixing  (z,  y)  in  Hr  results  in  t  new  pairs  (ot,  6{)  that  are  bad  in  Hr',  then  we  can  fix 
all  of  these  pairs  in  t  apf^ications  of  the  inductive  construction.  (Note  that  the  t  bad  pairs 


22 


do  not  interfere  with  each  other;  since  all  of  them  are  bad,  they  all  fall  under  one  of  the  first 
11  cases.  Therefore,  fixing  one  pair  (a^,  6,  )  either  fixes  another  pair  (o^,  6^)  or  does  not  affect 

Thxis,  the  number  of  bad  pairs  in  Hr  can  be  reduced  by  at  least  one  in  some  finite  niunber 
of  applications  of  the  inductive  construction  given  in  the  base  case.  Furthermore,  this  number  is 
bounded  by  n. 

'Iherefore,  we  can  construct  a  history  Hr'  with  fewer  than  k  bad  pairs  such  that  t'  =p  r.  firom 
the  Induction  Hypothesis,  there  is  a  run  r"  that  satisfies  FS2  such  that  r'  =p  r";  therefore,  r  =p  r". 

□ 


A3  Proof  of  Theorem  6 

Theorem  6  (Vr :  r  □8FS2b)  =>  (Vr ;  r  DW). 


We  will  show  that  (3r :  r  ^  0“’VV)  ^  (3r ;  r  |=  -i0sFS2b).  To  do  this,  we  first  assume  that 
W  does  not  hold  in  some  state  of  r,  i.e.,  that  it  is  possible  for  k  fiuliunes  to  be  detected  such  that 
the  quorum  sets  for  those  detections  have  an  empty  intersection.  We  then  show  that  using  this 
assumption,  a  run  can  be  constructed  in  which  there  is  a  Jb-cyde  in  the  failed-before  relation. 

We  divide  then  processes  in  Pinto  A  sets  5o,...,5fc_i  such  that  for  0<  t  <  ib-l,»€  5,;thatis, 
processes  Othrou^l;- 1  are  in  sets  5o  through  and  the  rest  of  the  processes  are  distributed 
among  So  througfi  Sk-\. 

Consider  the  following  scenario.  For  aU  t :  0  <  t  <  (ib  ~  1): 

1.  Process  i  suspects  the  failiue  (rf  process  i  0 1,  and  sends  the  message  SUSP^i^i  to  all  processes 
in  P.  The  messages  sent  to  the  processes  in  set  are  delayed  indefinitely. 


2.  As  a  result  of  Step  1,  process  t  receives  a  message  SUSPjeij  process  j  @  1  for  all 
j  hO  <  j  <  k  -  1,  where  @  is  subtraction  modulo  k.  Thus,  process  i  does  not  learn  that 
another  process  has  su^)ected  it  of  having  crashed. 


3.  Before  receiving  SU^^eiJ/  process  t  suspects  the  failure  of  process  j,  and  sends  SUSP, j  to 
all  processes  in  P.  Ihe  messages  sent  to  the  processes  in  set  are  delayed  behind  the 
previous  messages  (recall  that  interprocess  channels  are  FIFO).  Process  t  also  acknowledges 
any  SUSP  messages  with  ACK3U5P  messages. 

4.  Process  i  has  now  received  ACK3USPfc,j®t  messages  from  all  processes  1;  in  (J  Sj. 

j9«ei 

Let^i,iei=  (J  forallt : 0 <  I  <  A:- 1.  Noprocessin5,  isin^,j®i;inotherwords,forevery 

Jjtiej 


fc-i 

process  i  in  P,  there  is  some  quorum  set  of  which  t  is  not  a  member.  Therefore,  Q  =  0. 

t=0 
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Ruthennore,  by  definition  of  Qij  being  a  quonim,  every  process  j  has  received  enough  ack^usp 
messages  to  execute/iit7eif{(t0 1).  Wehave/iiiia/o(l), . .  .,failed^i^_2^(k  - 1),  and  failed^i^_^^{0),  which 
causes  a  i;-.ycle  in  the  foiled-befbre  relation.  O 


A.4  Proof  that  the  Protocol  of  Section  5  Implements  sFS2b 

Lemma9  Gixxn  the  protocol  of  Section  5,  then  [r  f=  35  =  {1,2,. (FAILEDi(2)AFAILED2(3)A---A 
FAILEDifc_i(jfe))]  =►  [39:  (send,(5,  "kfaUed")-^  send,(5,  "k  -  1  failed")-^  ■  •  send,(5,  “Ifailed”)) 
in  Hr]. 

Proof:  We  use  the  notation  SEND, (5,  m)  as  shorthand  for  (Vp  e  S :  SENDi(p,  m)). 

The  size  of  the  quora  are  sufficient  to  ensure  W,  by  Theorem  7.  By  W,  r  f=  3^ :  Vi,  j  €  5 : 
FAlLEDi(j)  =»  REC:Vi(9,  "j  failed")  =»  SEND, (5,  "j  failed").  We  prove  the  lemma  by  induction  on  k. 

Base  case  For  k  =  2,  the  proof  is  trivial  Let  k  =  3.  5  =  {1,2,3},  r  [=  faileDi(2)  a 
FAILED2(3),  and  r  |=  SEND,(S,  "2  failed")  A  SEND,(S,  "3  failed").  Assume  for  contradiction  that 
send, (5,  "2  failed")  send,(5,  "3  failed")  in  Hr-  Then,becausechannelsareFIFO,rBCP2(g,  "2failed") 
recD2(q,  "3  failed")  in  W, .  By  the  protocol  crashi  -*■  failed2(3)  in  Hr,  so  r  ^  ->FAILED2(3).  Therefore, 
it  must  be  the  case  that  sendq(S,  "3  failed")  — sendq(S,  "2  failed"). 

Induction  case  Assume  that  die  lemma  is  true  for  fc  =  /  -  1.  For  fc  =  /,  we  have  FAILEDi(2)  A 
FAIUED2(3)  a  •  •  •  a  FAlLEDj-t(l).  By  the  induction  hypothesis,  scnd,(5,  "1-1  failed") 
send, (5,  "2  failed")  in  7ir>  Assume  for  contradiction  that  send,  (5,  "1  - 1  failed")  — ►  send,(5,  "1  failed") 
in  Hr.  Then,  as  in  the  base  case,  recoi^\{q,  "1  —  1  failed")  -*^  recvi^\{q,  "1  failed"),  so  cnisli(_i  -♦ 
faiJedi_j(l)  in  Hr  and  r  ^  -iFAILED/_i(1).  Iherefore,  sendq{S,  "1  feiiled")  -►  send, (5,  "1-1  failed") 
inHr.  ° 

The  quorum  size  for  each  failure  detection  is  sufficient  to  guarantee  >V.  Assume  for  contra¬ 
diction  that  the  failed-before  relation  is  not  acydic  Then  r  |=  35  =  {1,  ...,*} :  faileDi(2)  a  •  •  •  a 

FAlLEDifc_i(]fc)  A  FAlLEDfc(l).  By  Lemma  9, 3g:  send,(5,  "1  failed")  -♦  send,(5,  "k  failed")  - - 

send,(5,  "2  failed")  in  W,.  Thus,  recvi{q,  "1  failed")  recri(g,  "2 failed")  in  Wr,  crash]  — 

in  Hr,  and  r  |=  -»FAILEDi(2).  □ 
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